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Abstract: I describe the current status of MCnet tools for validating the performance 
of event generator simulations against data, and for tuning their phenomenological free 
parameters. For validation, the Rivet toolkit is now a mature and complete system, with 
a large library of prominent benchmark analyses. For tuning, the Professor system has 
recently completed its first tunes of Pythia 6, with substantial improvements on the existing 
default tune and potential to greatly aid the setup of new generators for LHC studies. 
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1. Introduction 

It is an inevitable consequence of the physics approximations in Monte Carlo event genera- 
tors that there will be a number of relatively free parameters which must be tweaked if the 
generator is to describe experimental data. Such parameters may be found in most aspects 
of generator codes, from choices of Aqcd and p± cutoff in the perturbative parton cas- 
cade, to the non-perturbative hadronisation process. These latter account for the majority 
of parameters, since the models are deeply phenomenological, typically invoking a slew of 
numbers to describe not only the kinematic distribution of p± in hadron fragmentation, but 
also baryon/meson ratios, strangeness and {rj, r]'} suppression, and distribution of orbital 
angular momentum [1-4]. The result is a proliferation of parameters — of which between 
0(10-30) may be of particular importance for physics studies. 

Apart from rough arguments about their typical scale, these parameters are freely- 
floating: they must be matched to experimental data for the generator to perform well. 
Additionally, it is important that this tuning is performed against a wide range of experi- 
mental analyses, since otherwise parameters to which the selective analyses are insensitive 
will wander freely and may drive unconsidered observables to bad or even unphysical places. 
This requires a systematic and global approach to generator tuning: accordingly, I will sum- 
marise the current state of tools for systematically validating and tuning event generator 
parameters, and the first results of such systematic tunings. 

2. Validation tools: Rivet 

The Rivet library is a successor to the successful HERA-oriented generator analysis library, 
HZTool [5]. Like its predecessor, the one library contains both a library of experimental 
analyses and tools for calculating physical observables. It is written in object-oriented C++ 
and there is strong emphasis on the following features: 

• strict generator-independence: analyses are strictly performed on HepMC [6] event 
record objects with no knowledge of or ability to influence the generator behaviour; 
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• experimental reference data files are included for each standard analysis, and are used 
to ensure that analysis data binnings match their experimental counterparts as well 
as for fit comparisons; 

• computational results are automatically cached for use between different analyses, 
using an infrastructure mechanism based on "projection" classes; 

• clean, transparent and flexible programming interface: while much of the complexity 
is hidden, analyses retain a clear algorithmic structure rather than attempting to 
hide everything behind "magic" configuration files. 

The "projection" objects used to compute complex observables are now a fairly complete 
set: 

• various ways to obtain final state particles: all, charged only, excluding certain par- 
ticles, with p± and rapidity cuts, etc.; 

• event shapes: sphericity, thrust, Parisi C & D parameters, jet hemispheres; 

• jet algorithms: CDF and D0 legacy cones, Durham/JADE, and k±, anti-A;j_, SIS- 
Cone, CDF "jetclu" etc. from Fast Jet [7]; 

• miscellaneous: jet shapes, isolation calculators, primary and secondary vertex finders, 
DIS kinematics transforms, hadron decay finder, etc. 

The set of standard analyses has also grown with time and is now particularly well- 
populated with analyses from the LEP and Tevatron experiments: 

• LEP: Aleph and Delphi event shape analyses; Aleph, Delphi and PDG hadron 
multiplicities, strange baryons; Delphi and Opal b-fragmentation analyses; 

• Tevatron: CDF underlying event analyses (from 2001, 2004 k 2008); CDF and D0 
E W boson p±_ analyses; CDF and D0 QCD colour coherence, jet decor relation, jet 
shapes, Z+jets, inclusive jet cross-section; 

• HERA: HI energy flow and charged particle spectra; ZEUS dijet photoproduction. 

In addition, users can write their own analyses using the Rivet projections without needing 
to modify the Rivet source, by using Rivet's plugin system. We encourage such privately- 
implemented analyses to be submitted for inclusion in the main Rivet distribution, and 
would particularly welcome QCD analyses from HERA, b-factory and RHIC p-p experi- 
ments. 

While Rivet is primarily a library which can be used from within any analysis frame- 
work (for example, it is integrated into the Atlas experiment's framework), the primary 
usage method is via a small executable called rivetgun. This provides a frontend for reading 
in HepMC events from plain text dump files and also for running generators "on the fly" 
via the AGILe interface library. This latter approach is particularly nice because there is 
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no need to store large HepMC dump files and the corresponding lack of file I/O speeds up 
the analysis by a factor ~0(1O). In this mode, Rivet is ideal for parameter space scans, 
since generator parameters can be specified by name on the rivetgu n command line and ap- 
plied without recompilation. AGILe currently supports API-level interfaces to the Fortran 
Herwig 6 [2] and Pythia 6 [1] generators (combined with the AlpGen [8] MLM multi-jet 
merging generator, the Charybdis black hole generator [9], and the Jimmy hard underly- 
ing event generator [10] for Herwig), plus the C++ generators Herwig++ [3], Sherpa [4], 
and Pythia 8 [11]. 

At the time of writing, the current version of Rivet is 1.1.1. The main framework 
benefits of the 1.1.x series over 1.0.x are a safer and simpler mechanism for handling 
projection objects (massively simplifying many analyses), better compatibility of the AG- 
ILe loader with the standard LCG Genser packaging and a large number of new and 
improved analyses and projections. A "bootstrap" script is provided for easy setup. Any- 
one interested in using Rivet for generator validation should first visit the website http: 
//projects .hepforge . org/rivet/. 

Rivet is now a stable and powerful framework for generator analysis and we are looking 
forward to its increasing role in constraining generator tunings for background modelling in 
LHC high-p± physics. Future versions will see improvements aimed at high-statistics vali- 
dation simulations, such as histogramming where statistical error merging is automatically 
correct, as well as the addition of more validation analyses. 

3. Tuning tools: Professor 

While Rivet provides a framework for comparing a given generator tuning to a wide range 
of experimental data, it provides no intrinsic mechanism for improving the quality of that 
tune. Historically, the uninspiring task of tuning generator parameters to data "by eye" 
has been the unhappy lot of experimental researchers, with the unsystematic nature of 
the study reflecting that significant improvements in quality of both life and tuning would 
have been possible. This call for an automated and systematic approach to tuning is taken 
up by a second new tool: Professor. This is written in Python code as a set of factorised 
scripts, using the SciPy numerical library [12] and an interface to rivetgun. 

The rough formalism of systematic generator tuning is to define a goodness of fit 
function between the generated and reference data, and then to minimise that function. 
The intrinsic problem is that the true fit function is certainly not analytic and any iterative 
approach to minimisation will be doomed by the expense of evaluating the fit function at 
a new parameter-space point: this may well involve ten or more runs of the generator with 
200k-2M events per run. Even assuming that such runs can be parallelised to the extent 
that only the longest determines the critical path, an intrinsically serial minimisation of 
0(1000) steps will still take many months. This is clearly not a realistic strategy! 

The Professor approach, which is the latest in a lengthy but vague history of such 
efforts [13,14], is to parameterise the fit function with a polynomial. In fact, since the fit 
function itself is expected to be complex and not readily parameterisable, there is a layer of 
indirection: the polynomial is actually fitted to the generator response of each observable 
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bin, MCfe to the changes in the n-element parameter vector, p. To account for lowest-order 
parameter correlations, a second-order polynomial is used, 



mc 6 (p ) « /<»> (p) = 4 b) + E + E ^ 4 



(3-1) 



where the shifted parameter vector p ' = p—po , with p*o chosen as the centre of the parameter 
hypercube. A nice feature of using a polynomial fit function, other than its general-purpose 
robustness, is that the actual choice of the po is irrelevant: the result of a shift in central 
value is simply to redefine the coefficients, rather than change the functional form, but 
choosing a central value is numerically sensible. 

The coefficients are determined by randomly sampling the generator from TV parameter 
space points in an n-dimensional parameter hypercube defined by the user. Each sampled 
point may actually consist of many generator runs, which are then merged into a single 
collection of simulation histograms. A simultaneous equations solution is possible if the 
number of runs is the same as the number of coefficients between the n parameters, i.e. 
N = N^l = (2 + 3n + n 2 )/2. However, using this minimum number of runs introduces a 
systematic uncertainty, as we certainly do not expect the bin MC response to be a perfect 
polynomial. Here we are helped by the existence of the Moore-Penrose pseudoinverse: a 
generalisation of the normal matrix inverse to non-square matrices with the desirable fea- 
ture that an over-constrained matrix will be inverted in a way which gives a least-squares 
best fit to the target vector. Even more helpful is that a standard singular value decom- 
position (SVD) procedure can be used to deterministically implement the pseudoinverse 
computation. Hence, we phrase the mapping on a bin-by-bin basis from coefficients C to 
generator values V as PC = V, where P is the parameter matrix to be pseudo-inverted. 
For a two parameter case, parameters E {x,y}, the above may be explicitly written as 



where the numerical subscripts indicate the N generator runs. Note that the columns of P 

(2) 

include all -/V" mm = 6 combinations of parameters in the polynomial, and that P is square 
(i.e. minimally pseudo-invertible) when N = N^^. Then C = I[P] V, where Z is the 
pseudoinverse operator. 

Now that we have, in principle, a good parameterisation of the generator response to 
the parameters, p, for each observable bin, b, it remains to construct a goodness of fit (GoF) 
function and minimise it. We choose the x 2 function, but other GoF measures can certainly 
be used. Since the relative importance of various distributions in the observable set is a 
subjective thing — given 20 event shape distributions and one charged multiplicity, it is 
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Figure 1: (a) Parameter space line scan in x 2 , showing the agreement between Professor's predicted 
values (blue lines) and the true values (red dots), (b) Pythia 6 b- fragmentation functions, showing 
the improvements obtained using Professor (red) to tune the Bowler parameterisation against the 
default (blue). 



certainly sensible to weight up the multiplicity by a factor of at least 10 or so to maintain 
its relevance to the GoF measure — we include weights, wq, for each observable, O, in our 
X 2 definition: 

x (p) = 2^wo2_^ ' ( 3 - 3 ) 

o beo b 

where TZb is the reference value for bin b and the total error A& is the sum in quadrature of 
the reference error and the statistical generator errors for bin b — in practise we attempt 
to generate enough data that the MC error is much smaller than the reference error for all 
bins. 

The final stage of our procedure is to minimise this parameterised x 2 function. It is 
tempting to think that there is scope for an analytic global minimisation at this order of 
polynomial, but not enough Hessian matrix elements may be calculated to constrain all 
the parameters and hence we must finally resort to a numerical minimisation. We have 
implemented this in terms of minimisers from SciPy and also PyMinuit [15], with the 
latter's initial parameter space grid scan making it our preferred choice. 

Finally, on obtaining a predicted best tune point from Professor, it is prudent to check 
the result. This can be done directly with rivetgun, and Professor also has a line scan feature 
which allows scans along arbitrary straight lines in parameter space, which is useful to verify 
that the x 2 behaves as interpolated and to explicitly compare default tunes to predicted 
tunes. Such a line scan can be seen in Fig. |l](a). We have explicitly checked the robustness 
of the polynomial and the random distribution of sampling points against various skewed 
test distributions and the behaviour is robust. We have also found it to be useful to over- 
sample by a considerable fraction, and then to perform the x 2 minimisation for a large 

(n) 

number of distinct run-combinations, < N tunc < N, which gives a systematic control 
on interpolation errors and usually a better performance than just using iVt une = -/V. 1 

1 Note that the tuning runs need a significant degree of variation, i.e. Ar tuae *C N for most of the tune 
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The focus in testing and commissioning the Professor system has until recently been 
focused on Pythia 6 tunes against LEP data [16]. Here we were able to interpolate and 
minimise up to 10 parameters at a time for roughly 100 distributions, but beyond this the 
minimisation time became large and we were less happy with the minima. Eventually we 
decided to split the tuning into a two-stage procedure where flavour-sensitive fragmentation 
parameters were tuned first to provide a base on which to tune the semi-factorised kinematic 
parameters of the shower and hadronisation. The result has been a dramatic improvement 
of the Pythia 6 identified particle multiplicity spectra, without losing the event shape 
descriptions (originally tuned by Delphi's version of the same procedure), and a major 
improvement of the b-fragmentation function as seen in Fig. |l](b). 2 This tune will be 
adopted as the default parameter set for the next release of Pythia 6. 

4. Conclusions 

To conclude, the situation is looking positive for MC generator tuning at present: the Rivet 
and Professor tools are now in a state where they can be used to achieve real physics goals 
and the Pythia 6 tune described here (using both tools) has been a significant success. 
Development plans in the near future are very much aimed at getting the same tuning 
machinery to work for hadron collider studies, in particular initial state radiation (ISR) 
and underlying event (UE) physics. We aim to present tunes of C++ generators to LEP 
data shortly, along with first studies of interpolation-based tunes to CDF underlying event 
data. Finally, we are keen to constrain fragmentation and UE hadron physics for the LHC, 
using b-factory, RHIC and early LHC data. 
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